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EXECUTIVE  SUMMARY 


This  report  provides  the  first  results  of  laboratory  testing  of  the  Computer-Assisted  Data 
Acceptance  tools  for  the  acceptance  of  Contractor-provided  CALS  raster  type  1  data.  The 
testing  to  date  has  only  been  performed  with  legacy  data  obtained  from  the  DSREDS  and 
EDCARS  sites  that  was  converted  to  meet  the  CALS  standards.  It  has  been  difficult  to  obtain 
data.  Thanks  to  the  efforts  of  the  CALS  Test  Network  Office  (CT'NO),  Lawrence  Livermore 
Laboratories,  and  the  Communication  Electronics  Command  (CECOM)  Concurrent 
Engineering  Directorate  three  set  of  data  were  obtained. 

Each  of  the  three  sets  of  contractor  data  were  loaded,  evaluated  by  the  CADA  tools  and  the 
results  analyzed  as  reported  herein.  A  great  deal  of  difficulty  was  encountered  in  applying 
the  CADA  tools  to  the  IBM  and  Hughes  data  since  it  was  not  of  high  quality  and  of  a  format 
that  had  not  been  previously  analyzed.  The  data  from  IBM  and  Hughes  consisted  mainly  of 
technical  illustrations  with  no  borders.  A  high  rejection  rate  was  obtained  on  these  images 
because  the  current  CADA  algorithms  work  best  with  engineering  drawings  that  conform  to 
ANSI  standards  and  not  on  technical  illustrations.  The  ID  recognition  portion  was  not  tested 
on  these  images  because  they  have  no  borders. 

The  data  obtained  from  CECOM  was  from  a  contract  with  Magnavox  and,  of  the  three  sets, 
was  the  most  representative  of  the  data  expected  to  be  received  by  the  DSREDS,  EDCARS, 
and  JEDMICS  repositories.  About  90%  of  these  images,  consisting  of  engineering  drawings, 
parts  lists,  and  associated  lists  were  accepted  by  the  CADA  image  analysis  software. 

However,  a  large  percentage  of  these  images  (about  75%)  were  rejected  in  the  ID  recognition 
portion  due  to  different  reasons  documented  in  this  report. 

The  results  indicate  that  further  work  is  needed  to  improve  the  accuracy  of  the  CADA  tools. 
For  image  quality  analysis,  it  is  recommended  that  additional  work  be  performed  with  the 
Magnavox  data  and  with  the  local  CECOM  QA  inspectors  in  establishing  general,  objective 
criteria  for  image  acceptance  and  tuning  the  CADA  software  to  match  with  those  criteria.  A 
test  plan  has  been  prepared  to  implement  this  next  step  in  the  use  of  the  CADA  tools  to 
accept  contractor  data. 

For  ID  recognition,  improvement  is  required  in  all  the  three  stages  of  pre-processing, 
recognition,  and  post-processing.  Work  is  underway  in  each  of  these  areas  and  a  test  report 
will  be  submitted  showing  the  results  of  the  laboratory  testing  of  the  Magnavox  contractor 
data  as  it  compares  to  manual  testing  performed  by  CECOM  DSREDS  QA  personnel. 
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Disclaimer 


The  use  of  trade  names  in  this  document  or  discussion  of  a  particular  product  does  not 
constitute  an  endorsement  or  approval  of  the  use  of  such  commercial  equipment.  This 
document  may  not  be  cited  for  the  purpose  of  advertisement 
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1  INTRODUCTION 

This  report  provides  the  first  documented  results  of  the  laboratory  tests  of  Computer-aided 
Acquisition  Logistic  Support  (CALS)  Contractor-provided  raster  data  and  the  Computer- 
Assisted  Data  Acceptance  (CADA)  software  tools.  Testing  was  performed  on  a  Sun  platform. 
Field  operational  testing,  within  the  Air  Force  and  Army,  of  contractor  provided  CALS  raster 
data  via  use  of  the  CADA  tools,  on  specified  platforms,  will  be  the  next  step  in  the  CADA 
tool  development  testing  process. 

1.1  Background 

The  Army  has  been  designated  by  the  Department  of  Defense  (DoD)  CALS  as  the  lead 
Service  to  develop  and  test  procedures  for  the  acceptance  of  CALS  digital  data.  The  CALS 
Technology  Center  (CTC)  located  at  Ft.  Monmouth,  NJ,  is  the  designated  test  bed  for  the 
Army  Service.  Program  Manager  (PM)  Joint  CALS  (JCALS),  under  the  direction  of  the 
CALS  Test  Network  Office  (CTNO),  has  developed  CADA  procedures  for  accepting 
engineering  data  in  CALS  raster  format.  The  CADA  procedures  depend  on  tools  and 
techniques  that  can  automate  the  analysis  of  the  digital  data.  The  two  key  areas  identified  for 
automating  the  acceptance  of  CALS  digital  data  are  the  quality  of  the  Image  data  and  the 
accuracy  and  quality  of  the  Identification  Data  (ID). 

Previous  reports  have  documented  the  testing  of  the  CADA  tools  via  use  of  laboratory 
generated  test  data  and  a  representative  sample  of  some  6000  images  obtained  from  the  Army 
DSREDS  and  Air  Force  EDCARS  repositories.  Most  of  this  testing  was  to  evaluate  the 
different  image  algorithms  individually  and  in  various  combinations.  A  subset  of  image 
algorithms  were  integrated  and  through-put  performance  testing  was  conducted  on  a  Sun 
platform  and  a  PC  platform.  The  verification  of  the  ID  inside  the  CALS  header  with  the 
identification  within  the  image  area  of  the  engineering  drawing  was  then  tested  with  pre¬ 
processing  software  and  with  OCR/ICR  engines  for  both  accuracy  and  speed.  These  reports 
entitled  Testing  Advanced  Techniques,  dated  12  July  1991  and  CADA  Performance  Test 
Report,  dated  9  November  1992  are  available  from  the  CALS  Test  Network  Office  (CTNO), 
Dayton  OH  or  the  PM  JCALS  office,  Ft.  Monmouth,  NJ. 

1.2  Purpose 

This  report  presents  the  Laboratory  Acceptance  Testing  of  Contractor-provided  CALS  data  via 
the  use  of  automated  software  (CADA)  tools.  The  main  objectives  of  the  tests  were: 

1.  perform  and  document  manual,  100%  quality  assurance  (QA),  acceptance 
testing  of  Contractor-provided  CALS  data  at  the  PM  JCALS,  CALS 
Technology  Center  (CTC)  by  CADA  personnel; 

2.  perform  automated  CADA  laboratory  acceptance  tests  of  the  data  that  had  been 
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evaluated  manually  by  CTC  CADA  personnel; 

3.  provide  a  comparison  of  the  manual  and  automated  (CADA)  test  results;  and 

4.  provide  a  summary  of  the  analysis  with  recommendations. 

1.3  Scope 

The  scope  of  testing  involves  evaluating  the  manual  and  automated  acceptance  of  Contractor- 
provided  CALS  raster  data  within  the  CTC  laboratory.  The  extent  of  the  tests  included: 

1.  obtaining  the  contractor  data  from  the  CECOM  Concurrent  Engineering 
Directorate  and  coordinating  the  manual  tests  within  the  CTC  by  CADA 
personnel; 

2.  obtaining  additional  contractor  data  from  the  CTN  Lawrence  Livermore 
Laboratories  and  performing  manual  and  automated  (CADA)  tests  within  the 
PM  JCALS,  CALS  Technology  Center  (CTC)  laboratory  environment; 

3.  performing  the  tests  and  comparing  the  results;  and 

4.  preparing  a  report  of  the  testing  results. 
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2  TESTING  OUTLINE 

2.1  Location 

Laboratory  Contractor  data  testing,  using  CADA  tools,  analysis  of  the  test  results,  and 
preparation  of  this  test  report  was  performed  at  the  PM  JCALS  CALS  Technology  Center 
(CTC)  at  Fort  Monmouth,  New  Jersey,  for  the  CALS  Test  Network  Office. 


2.2  Test  Plans 

The  Test  Plan  which  is  located  in  Appendix  B  of  this  document,  was  prepared  to  facilitate 
concurrent  testing  of  a  common  set  of  contractor  data  in  a  laboratory  environment  and  by 
users  in  a  field  environment  Difficulty  in  loading  the  CALS  data  at  the  field  DSREDS  site 
delayed  the  field  testing  so  the  Test  Plan  was  changed  to  conduct  the  tests,  at  this  time,  in  the 
CTC  laboratory  environment  only. 

The  implementation  of  the  Test  Plan  was  changed  to  include  only  the  PM  JCALS  CTC 
participation  defined  in  Section  4.2  of  the  Test  Plan  and  manual  QA  by  CADA  personnel. 
This  effort  is  based  on  the  results  of  these  tests. 
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3  LABORATORY  CONTRACTOR  DATA  TEST  PARAMETERS 

Dates: 

Actual  Testing:  November  and  December  1992 
Evaluators: 

PM  JCALS  Technology  Center  Personnel 
PM  JCALS,  SFAE-PS-CAL-T 
Ft.  Monmouth,  New  Jersey 

ACCURATE  Information  Systems,  Inc.  Personnel 
Meridian  Center  One, 

Two  Industrial  Way, 

Eatontown,  New  Jersey  07724 

Data  Types: 

MIL-R-28002  Raster  Type  1  binary  data  on  CALS  formatted  Magnetic  Tape  media 
from  the  following  contractors: 

Hughes 

IBM 

Magnavox 
System  Description: 

PM  JCALS  CADA  Platform  Hardware:  Sun  IPX;  48MB  RAM,  1.3GB  H.  Disk, 

HP  Tape  Drive  and  a  PC  Gateway  2000, 
50MHZ  -  486  DX2  with;  16MB  RAM, 
340MB  SCSI  Hard  Disk 

PM  JCALS  CADA  Platform  Software:  SunOS  4.1.1,  Sun  NewsServer  -X11R4 

compliant  MOTIF  Window  Manager  1.1 
and  on  the  PC:  SCO  UNIX  Open  Desktop 
2.0,  X11R4  Server,  MOTIF  Window 
Manager  1.1 

PM  JCALS  CADA  Software:  CADA  Tools  Ver.  0.5  with:  XVT 

Portability  Toolkit  Ver  3.0  UniSoft’s  PC 
Imaging  Library  Ver.  5.1,  CTN’s  Tape 
Tool  Ver  1.2.8  NestorReader  PC  Ver.  1.2 
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4  LABORATORY  CONTRACTOR  DATA  TEST  PROCEDURE 
SUMMARY 

The  availability  of  CALS  Contractor  data  has  been  a  problem.  A  number  of  Government 
agencies  have  cooperated  since  June  of  1992  in  an  attempt  to  obtain  data.  We  were  able  to 
obtain  three  magnetic  tapes  through  the  efforts  of  the  CALS  Test  Network  Office  (CTNO), 
the  Lawrence  Livermore  National  Laboratory,  and  the  Army  CECOM  Concurrent  Engineering 
Directorate  at  Ft  Monmouth,  New  Jersey.  The  Contractor  data  sources  and  the  approach  in 
conducting  the  tests  within  the  CTC  of  PM  JCALS  are  described  herein. 

4.1  Test  Data  Sources 

Table  1  provides  a  summary  of  the  source  of  the  CALS  contractor  data  and  the  organizations 
responsible  for  obtaining  the  data. 


Table  1.  Test  Data  Sources 


Media  /  Qty 

Contractor 

Responsible  Organizations 

Hughes 

Lawrence  Livermore  Laboratories 

IBM 

Lawrence  Livermore  Laboratories 

Mag.  Tape  /  1 

Magnavox 

Army  CECOM  Concurrent  Engineering 
Directorate 

4.2  Testing  Approach 

The  approach  taken  in  testing  the  three  sets  of  contractor  data  was  to  load  the  data  onto  the 
Sun  platform,  conduct  CADA  evaluations  for  image  quality  and  identification  data  (ID) 
accuracy  and  then  analyze  the  results.  Three  sets  of  image  evaluation  parameters  were 
applied:  high  quality,  medium  quality,  and  low  quality,  and  the  results  recorded.  For  each 
set  of  image  parameters  both  image  and  ID  evaluations  were  performed.  The  ID  evaluations 
were  only  performed  for  those  images  that  were  accepted  by  image  evaluations.  The  image 
and  ID  results  were  then  analyzed  and  the  report  prepared. 

4.2.1  Acquire  Contractor  Data 

Three  sets  of  contractor  data  were  used  in  evaluating  the  CADA  tools.  Two  sets  were 
obtained  by  Lawrence  Livermore  Laboratories  from  Hughes  and  IBM.  A  third  set  was 
provided  by  the  CECOM  Concurrent  Engineering  Directorate  that  was  prepared  by 
MagnaVqx. 
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The  Hughes  data  contains  217  images  that  consist  of  mosdy  Notices  of  Request  (NORs)  and 
Engineering  Change  Requests  (ECRs).  There  are  no  title  blocks  or  borders  for  most  of  the 
images,  and  the  quality  is  poor. 

The  IBM  tape  consists  of  71  images  that  are  technical  illustrations  with  no  borders  and  no 
identification  data.  Most  of  the  images  are  readable  but  have  characteristics  different  from 
engineering  quality  data,  causing  a  substantial  number  of  rejects. 

The  Magnavox  tape  contains  228  images  consisting  of  engineering  drawings,  parts  lists  and 
data  lists.  Most  of  these  images  meet  the  specifications  for  format;  however,  many  of  the 
images  are  light  and  have  faded  areas. 

4.2.2  Load  and  Evaluate  CALS  Media  Format 

The  Contractor-provided  CALS  magnetic  tapes  were  loaded  onto  the  CADA  platform  and  the 
CALS  format  was  validated.  Some  problems  were  found  initially,  for  example,  the  tapes 
provided  by  Magnavox  contained  header  record  formats  that  did  not  conform  to  the  MIL-D- 
1840A  standard.  Magnavox  corrected  the  problem  and  provided  new  CALS  magnetic  tapes. 

4.2.3  Conduct  CADA  Image  Testing 

All  the  contractor  tapes  were  unloaded  using  the  CADA  tool.  The  tools  were  then  used  to 
evaluate  the  accuracy  of  ID  information  and  image  qualities.  Then  the  images  were  visually  • 
inspected  to  identify  false  rejection  and  false  acceptance  and  the  results  were  analyzed. 

4.2.3.1  IBM  Tests 

There  are  three  documents  on  the  IBM  tape,  each  consisting  of  an  SGML  set  and  a  set  of 
technical  illustrations.  Only  one  of  those  three  sets  of  technical  illustrations  was  tested.  Due 
to  decompression  errors,  the  other  two  sets  could  not  be  evaluated,  the  CADA  image 
evaluation  tools  were  utilized  to  check  the  image  quality.  Because  these  technical 
illustrations  contained  no  borders  and  no  identification  data,  ID  recognition  was  not 
performed. 

4.2 3.2  Hughes  Tests 

The  Hughes  tape  consists  of  20  sets  of  images;  most  of  them  are  NORs  and  ECRs,  and  there 
are  no  engineering  drawings.  Image  evaluation  was  performed  for  all  20  sets,  for  a  total  of 
217  image  files.  The  majority  have  no  title  block  and  no  border,  therefore,  ED  recognition 
was  not  used  on  these  sets. 

4.2.3.3  Magnavox  Tests 

There  are  50  sets  of  engineering  drawings  and  accompanying  documents  on  the  Magnavox 
tape.  The  engineering  drawing  sizes  vary  from  A  to  E  and  the  accompanying  documents 
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contain  Data  List,  Part  List  and  Associate  Lists.  All  of  the  228  image  files  were  tested  using 
CADA  image  evaluation  and  ID  recognition  tools.  Three  sets  of  parameters  were  used  to 
evaluate  different  levels  of  image  quality  (high,  medium  and  low  quality). 

After  evaluating  the  data  using  CADA  tools,  all  data  were  visually  inspected  to  verify  the  test 
results. 

4.2.4  Conduct  CADA  ID  Testing 

The  ID  recognition  portion  of  CADA  verifies  that  the  ID  data1  within  the  CALS  1840  header 
matches  the  corresponding  data  appearing  in  the  title  block  of  the  engineering  drawing.  The 
software  which  performs  this  is  contained  within  three  distinct  modules:  pre-processing, 
recognition,  and  post-processing. 

The  pre-processing  stage  consists  of  locating  the  border2  within  the  image,  cropping  the  title 
block  and  revision  block  portion  of  the  image  as  regions  of  interest  (ROI),  extracting  the 
components  within  the  ROIs  and  grouping  these  components  into  strings. 

These  strings  are  then  input  to  the  recognition  engine,  which  returns  the  recognized  ASCII 
results  along  with  their  confidence  values. 

The  recognition  results  are  then  processed  by  the  post-processing  unit  consisting  of  string 
matching  algorithms.  These  algorithms  consider  confusion  classes3  and  string  distances  in 
matching  the  CALS  header  with  the  recognized  information.  An  image  accept  or  reject 
decision  is  returned  based  on  these  comparisons. 

4.3  Analyze  CADA  Results 

4.3.1  Image  Analysis 

CADA  image  analysis  techniques  were  applied  on  the  three  sets  of  contractor  data  with  the 
following  parameter  settings. 

Run  length  ratio  <=  14 
Approximate  black  orphan  ratio  <=  0.024 
Approximate  white  orphan  ratio  <=  0.430 
Peak  tile  noise  <=165 


1  ID  data  consists  of  the  following  fields:  image  size,  drawing  number,  cage  code,  sheet  number,  and  revision 
letter. 

2  The  border  in  an  engineering  drawing  is  used  as  a  reference  frame  within  the  ID  software.  Hence,  failure  to 
locate  the  border  results  in  an  ID  reject  and  further  ID  processing  is  terminated. 

3  Similarly  shaped  characters  are  often  misinterpreted  (e.g.,  1,  i,  and  I  can  be  confused)  by  the  recognition 
engine.  Such  characters  are  termed  "confusion  classes"  and  have  to  be  accounted  for  in  the  interpretation  process. 
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The  results  are  summarized  in  the  tables  2  through  4.  Detailed  results  can  be  found  1ft 
Appendix  A. 


Table  2.  IBM  Results 


Total  number  of  images 

71 

Total  number  of  images  accepted  by  CADA 

23 

Total  number  of  images  rejected  by  CADA 

48 

Total  number  of  images  accepted  by  visual  inspection 

63 

Total  number  of  images  rejected  by  visual  inspection 

8 

Total  number  of  images  falsely  accepted  by  CADA 

0 

Total  number  of  images  falsely  rejected  by  CADA 

40 

Percentage  of  total  images  falsely  accepted  by  CADA 

0 

Percentage  of  total  images  falsely  rejected  by  CADA 

56.3 

Table  3.  Hughes  Results 


Total  number  of  images 

217 

Total  number  of  images  accepted  by  CADA 

68 

Total  number  of  images  rejected  by  CADA 

149 

Total  number  of  images  accepted  by  visual  inspection 

141 

Total  number  of  images  rejected  by  visual  inspection 

76 

Total  number  of  images  falsely  accepted  by  CADA 

5 

Total  number  of  images  falsely  rejected  by  CADA 

78 

Percentage  of  total  images  falsely  accepted  by  CADA 

2.3 

Percentage  of  total  images  falsely  rejected  by  CADA 

35.9 
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Table  4.  Magnavox  Results 


Total  number  of  images 

228  I 

Total  number  of  images  accepted  by  CADA 

195 

Total  number  of  images  rejected  by  CADA 

33 

Total  number  of  images  accepted  by  visual  inspection 

220 

Total  number  of  images  rejected  by  visual  inspection 

8 

Total  number  of  images  falsely  accepted  by  CADA 

2 

Total  number  of  images  falsely  rejected  by  CADA 

27 

Percentage  of  total  images  falsely  accepted  by  CADA 

0.87 

Percentage  of  total  images  falsely  rejected  by  CADA 

The  high  percentage  of  false  rejects  in  the  case  of  IBM  and  Hughes  data  is  due  to  the  fact 
that  the  data  in  those  sets  consists  mainly  of  technical  illustrations  and  not  engineering 
drawings.  CADA  techniques  work  best  with  engineering  drawings  conforming  to  ANSI 
standards  for  formats  and  line  widths. 

4.3.2  ID  recognition  analysis 

Magnavox  contractor  data  images  which  passed  image  analysis  were  input  to  the  ID 
recognition  portion  of  CADA.  The  results  of  this  analysis  will  be  used  as  a  feedback  to 
improve  various  components  of  ID  recognition.  The  results  are  listed  in  Table  5. 


Table  5.  ID  Recognition  Analysis  Results 


Total  images 

Image  rejects 

Assoc.  list 

ID  success 

ID  error 

228 

44 

12 

42 

130 

As  shown  in  Table  5,  44  of  the  total  228  images  were  rejected  by  the  image  analysis 
software.  Among  the  images  which  passed  image  analysis,  12  were  found  to  be  associated 
lists.  These  images  are  not  considered  because  the  current  ID  software  operates  only  on 
engineering  drawing  images.  As  a  result,  172  valid  engineering  drawings  were  input  into 
the  ID  recognition  software. 

Of  these  172  images,  42  were  ID  classification  successes  (they  were  either  correctly 
accepted  or  correctly  rejected  based  on  the  criteria  outlined  in  section  4.2).  It  should  be 
noted  that  there  were  no  false  accepts  among  the  130  ID  errors,  as  shown  in  Table  6. 
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Table  6.  Breakup  of  ED  Errors 


Total  ID  errors 

False  rejects 

False  accepts  1 

130 

130 

0  ] 

The  ID  errors  were  then  divided  among  the  following  sub-components:  pre-processing, 
recognition  engine,  and  post-processing.  These  results  are  listed  in  Table  7. 

The  pre-processing  is  further  categorized  into  errors  occurring  due  to  three  different 
reasons:  border  location,  broken  characters  in  the  key  id  text  data,  and  string  segmentation. 


Table  7.  ID  Errors  by  Causes 


Pre-processing 

Recognition 

Post-processing 

Border 

Broken 

Chars 

String 

Segment 

4 

18 

19 

83 

105 

It  should  be  noted  that  the  total  number  of  errors  in  different  categories  add  up  to  well 
over  130.  This  is  due  to  the  fact  that  in  a  single  image,  multiple  errors  may  occur  and 
these  are  counted  separately. 

Most  of  the  post-processing  errors  occur  in  interpreting  the  sheet  number.  Often  the  sheet 
number  is  far  away  from  its  label  and  the  current  criteria  are  very  strict  in  qualifying  it  to 
be  the  sheet  number  data.  Efforts  are  under  way  to  make  these  criteria  more  flexible  in 
order  to  accommodate  these  variations  and  thus  improve  the  acceptance  performance. 

The  errors  in  recognition  occur  mainly  due  to  the  characters  touching  each  other  or  the 
lines  of  the  enclosing  box.  The  reasons  can  be  attributed  to  scanning  problems  as  well  as 
fat  strokes  in  the  characters.  These  failures  cannot  be  helped  in  the  short  term.  Some  line 
thinning  or  skeletonization  algorithms  may  need  to  employed  as  part  of  the  pre-processing 
on  these  characters.  This  or  similar  efforts  are  being  researched  for  improved  pre¬ 
processing  performance. 

The  border  location  errors  in  the  pre-processing  are  mainly  due  to  the  resolution  of  the 
algorithm.  Any  further  increase  in  the  resolution  will  lead  to  increased  time  in  the 
calculations.  Some  character  fusing  algorithms  may  need  to  be  researched  in  order  to 
combat  the  problem  of  broken  characters.  The  string  segmentation  problem  can  be  tackled 
by  improved  string  forming  algorithms  in  pre-processing. 
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5  CONCLUSIONS 


5.1  Image  Analysis 

1.  The  current  CADA  techniques  work  best  on  engineering  drawings  conforming  to 
ANSI  standards.  They  do  not  work  well  on  technical  illustrations  and  text  data. 
Additional  techniques  should  be  incorporated  in  the  future  to  improve  the  results  on 
this  type  of  data. 

2.  So  far,  the  CADA  techniques  work  uniformly  on  all  images,  irrespective  of  their  type, 
size,  or  other  characteristics.  An  automated  tuning  of  parameters  based  on  the  type, 
size,  or  other  characteristics  of  images  can  possibly  improve  CADA  results. 

3.  CADA  techniques  are  designed  to  detect  general  flaws  like  presence  of  noise,  fading, 
discontinuities,  etc.  and  cannot  detect  specific  flaws  based  on  any  interpretation  of  the 
contents.  In  the  field,  CADA  can  be  most  useful  if  the  acceptance  criteria  are 
clearly  defined  in  general,  objective  terms  and  not  on  any  subjectiveness  or 
interpretation  of  contents. 

4.  The  Magnavox  data  provides  the  most  representative  range  of  data  among  the  three 
sets  tests  and  should  be  tested  further  with  CECOM  DSREDS  locally. 

5.2  ID  Analysis 

Most  of  the  post-processing  errors  occur  in  sheet  number  interpretation.  Often  the  distance 
between  the  sheet  label  and  data  exceeds  the  current  constraints  in  the  software.  Efforts  are 
underway  to  make  these  criteria  more  flexible  in  order  to  accommodate  these  variations  and 
thus  improve  the  acceptance  performance. 

The  errors  in  recognition  occur  mainly  due  to  touching  characters  in  the  raster  image.  These 
characters  not  only  touch  each  other,  but  often  overlap  the  enclosing  box.  The  reason  for  this 
problem  is  attributed  to  scanning  problems  as  well  as  thick  strokes  in  the  characters.  No 
short  term  solutions  exist  for  these  problems.  A  long  term  effort  involving  algorithm  analysis 
for  line  thinning  or  skeletonization  may  be  required. 

The  border  location  errors  in  the  pre-processing  section  are  due  to  the  resolution  of  the 
algorithm.  Any  further  increase  in  the  resolution  will  lead  to  increased  time  in  the 
calculations.  Some  character  fusing  algorithms  may  be  necessary  to  combat  the  problem  of 
broken  characters.  The  string  segmentation  problem  may  be  tackled  by  improved  criteria  in 
string  merging  algorithms. 
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6  RECOMMENDATIONS 

6.1  Image  Analysis 

1.  Further  analysis  should  be  conducted  to  establish  a  correlation  between  different  types 
of  images  and  CADA  parameter  settings.  For  certain  types,  new  techniques  such  as 
blob  analysis  should  be  explored. 

2.  Further  testing  should  be  done  in  collaboration  with  QA  inspectors  from  the  field  to 
establish  objective,  general  criteria  for  acceptance  and  tune  CADA  algorithms  to 
match  those  criteria. 

3.  Pursue  testing  of  the  Magnavox  data  with  CECOM  DSREDS  locally  in  accordance 
with  the  Test  Plan  included  in  Appendix  B,  and  prepare  a  test  report  containing  the 
results. 

6.2  ID  Analysis 

1.  The  following  specific  needs  for  improvement  in  the  subcomponents  of  the  ID 
software  emerged  during  the  contractor  data  tests: 

a)  sheet  label-data:  further  enhancements  to  allow  for  large  distances; 

b)  string  segmentation:  connected  component  grouping  criteria  requires  refinement; 

c)  line  thinning  algorithms:  required  for  addressing  problems  of  touching  characters; 
and 

d)  border  location:  requires  increased  resolution. 

2.  More  contractor  data  is  required  to  further  test  and  improve  the  ID  recognition 
algorithms.  Feedback  from  field  QA  operators  is  also  required  to  assist  with  future 
modifications  and  tuning. 
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APPENDIX  A 

CADA  TEST  RESULTS 
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APPENDIX  B 
CTC  TEST  PLAN 


CALS  Contractor  Data  Acceptance  Tests 
17  November  1990 


1.  Introduction  and  Background 

The  Project  Manager,  Joint  Computer-aided  Acquisition  and  Logistic  Support  (PM  JCALS) 
has  developed,  under  the  direction  of  the  CALS  Test  Network  Office  (CTNO),  Computer- 
Assisted  Data  Acceptance  /  Quality  Assurance  (CADA/QA)  tools  and  procedures  that  will 
automate  the  acceptance  of  CALS  formatted  raster  type  1  digital  data.  Integration  of  the 
CADA  tools  has  been  completed  and  operational  field  testing  of  the  tools  with  contractor 
furnished  data  is  scheduled  to  commence  in  early  1993.  In  preparation  for  the  field  tests, 
contractor  data  in  CALS  format  will  be  tested  within  the  PM  JCALS  CALS  Technology 
Center  (CTC). 

2.  Objective 

The  objective  of  the  laboratory  testing  of  CALS  contractor  data  is  to  test  the  tools  with 
different  parameters  in  order  to  obtain  the  best  QA  performance  on  "live"  data.  In  order  to 
achieve  this  objective,  the  Communication  Electronics  Command  (CECOM),  Concurrent 
Engineering  Directorate  (CED)  and  CECOM  Digital  Storage  and  Retrieval  Engineering  Data 
System  (DSREDS)  personnel  have  agreed  to  provide  CALS  contractor  data  and  participate  in 
the  testing  effort. 

3.  Standards  and  Specifications 

3.1  MIL-STD- 1 840A 

3.2  MIL-R-28002A 

4.  Procedures 

The  testing  of  CALS  contractor  data  is  a  cooperative  effort  between  the  ARMY  CECOM 
Concurrent  Engineering  Directorate  (CED),  CECOM  DSREDS  and  PM  JCALS  personnel. 
The  CED  will  obtain  the  contractor  data  on  a  CALS  formatted  Magnetic  Tape  and  provide  it 
to  PM  JCALS  who  will  load  the  data  and  return  the  Magnetic  Tape  to  CED.  CED  personnel 
will  give  the  Magnetic  Tape  to  CECOM  DSREDS  personnel  who  will  load  the  data  and 
manually  inspect  all  of  the  images  on  the  DSREDS  system.  PM  JCALS  will  evaluate  the 
contractor  data  with  the  CADA  tools  and  compare  the  results  with  the  results  obtained  from 
DSREDS.  The  CADA  parameters  will  then  be  changed  to  more  closely  match  the  results 
obtained  from  DSREDS.  The  following  scenario  will  be  followed  closely  in  order  to  meet  the 
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December  delivery  date. 

4.1  Concurrent  Engineering  Directorate  (CED)  Participation 

1.  Provide  comments  and  inputs  to  PM  JCALS  in  preparation  of  the  test  plan. 

2.  Obtain  the  magnetic  tape  from  the  contractor. 

3.  Deliver  the  magnetic  tape  to  the  CTC  for  format  verification  and  tape 
duplication. 

4.  After  format  verification,  obtain  the  original  magnetic  tape  from  the  CTC  and 
deliver  to  the  DSREDS  personnel. 

5.  Participate  in  the  review  of  the  CTC  and  DSREDS  testing  effort. 

6.  Review  the  draft  PM  JCALS  test  report  and  submit  comments. 

7.  Obtain  a  copy  of  the  PM  JCALS  final  test  report. 


4.2  PM  JCALS  CTC  Participation 

1.  Prepare  the  "draft"  test  plan. 

2.  Obtain  inputs  and  assistance  in  preparing  the  final  test  plan  from  the  CED  and 
DSREDS  personnel. 

3.  Obtain  magnetic  tape  from  the  CED,  load  image  data,  validate  MIL-STD-1840 
format,  prepare  a  data  list  and  return  the  magnetic  tape  and  a  data  list  to  CED. 

4.  Perform  the  CADA  testing.  Record  the  performance  times  to  load,  evaluate, 
inspect  and  print  the  results. 

5.  Obtain  the  inspection  results  from  DSREDS. 

6.  Analyze  the  compared  results,  modify  CADA  parameters  accordingly,  and  repeat 
the  CADA  tests.  Record  the  results  and  again  compare  with  DSREDS  manual 
inspection  results.  Repeat  the  CADA  tests  with  new  CADA  parameters,  as 
required,  to  obtain  the  results  that  best  agree  with  the  DSREDS  QA  analysis 
within  the  time  schedule  defined  in  Section  6. 

7.  Prepare  a  draft  test  report  and  review  the  results  with  CED  and  DSREDS 
personnel. 

8.  Finalize  the  test  report  and  submit  to  PM  JCALS/CTNO, 

4.3  CECOM  DSREDS  Participation 

1.  Provide  comments  and  inputs  to  PM  JCALS  in  preparation  of  the  test  plan. 

2.  Obtain  the  contractor  magnetic  tape  from  CED. 

3.  Load  the  magnetic  tape  onto  an  Intergraph  workstation  to  validate  the  MEL-STD 
-1840  format  and  convert  the  CALS  data  to  the  DSREDS  format.  Record  the 
loading  and  validation  elapsed  times. 

4.  Perform  100%  inspection  of  the  image  and  identification  data  using  the  DSREDS 
QA  workstation  and  existing  QA  procedures.  Record  the  time  to  accept  or  reject 
the  image  and  identification  data  for  accuracy  as  well  as  the  total  elapsed  time  to 
perform  the  manual  loading,  conversion  and  inspection  processes. 

5.  Submit  the  results  to  PM  JCALS  for  CADA  test  results  comparisons  and  input  to 


24 


Computer-Assisted  Data  Acceptance  (CADA)  Contractor  Test  Report 
the  test  report. 

6.  Review  the  draft  test  report  from  PM  JCALS  and  submit  comments. 

7.  Obtain  a  copy  of  the  PM  JCALS  final  test  report. 


5.  Facilities  and  Equipment 

1.  PM  JCALS  CTC  Sun  and/or  PC  Workstation  and  CADA  tools. 

2.  CECOM  DSREDS  facilities  and  equipment  required  to  perform  conversion  and 
inspection  of  CALS  data  submitted  on  magnetic  tape  media. 

6.  Schedule 

1.  The  CED  delivers  Contractor  data  on  MEL-STD-1840  formatted  Magnetic  Tape 
media  to  PM  JCALS.  -  Week  of  November  16,  1992. 

2.  PM  JCALS  validates  magnetic  tape  format  and  duplicates  tape.  November  20, 
1992  or  2  working  days  after  receipt  of  the  magnetic  tape  from  the  CED. 

3.  The  CED  delivers  the  Magnetic  Tape,  with  a  PM  JCALS  prepared  Data  List  to 
CECOM  DSREDS  one  working  day  after  receipt  from  the  PM  JCALS  CTC 
(Goal  November  24  1992). 

4.  PM  JCALS  completes  the  first  CADA  QA  Acceptance  tests  within  one  week 
after  receipt  of  the  magnetic  tape  (No  later  than  December  1,  1992). 

5.  CECOM  DSREDS  completes  100%  manual  QA  within _ week(s)  after  receipt 

of  the  magnetic  tape  -  No  later  than  December  7,  1992. 

6.  PM  JCALS  modifies  the  CADA  parameters,  repeats  the  CADA  tests,  documents 
results  in  a  draft  test  report  and  review  the  results  with  CED  and  DSREDS 
personnel.  (  By  December  14,  1992.) 

7.  PM  JCALS  prepares  a  final  internal  test  report  by  December  16,  1992. 

8.  PM  JCALS  management  receives  the  final  test  report  by  December  21,  1992. 


7.  Participants 

1.  PM  JCALS 

2.  Army  CECOM  CED 

3.  Army  CECOM  DSREDS 

4.  ACCURATE  Information  Systems,  Inc. 


